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ABSTRACT 



Validity generalization meta-analysis (VG) examines the 
extent to which the validity of an instrument can be transported across 
settings. VG offers correction and summarization procedures designed in part 
to remove the effects of statistical artifacts on estimates of association 
between criterion and predictor. By employing a random effects model, the 
variability of a distribution, "P, " of population parameters, "p, " is 
estimated. When the variance of this distribution is estimated to be small, 
validity is said to generalize across situations. It is common for an 
admissible validity study to contribute more than one correlation to a 
meta-analysis. The original VG meta-analysis (Pearlman, Schmidt, and Hunter, 
1980) located 3,368 validity coefficients in 698 studies. In addition, VG is 
often applied to instruments used to predict success on highly complex jobs. 
Such measures often have positively skewed distributions of predictor and 
criterion scores (Hunter, 1990) . This study used Monte Carlo simulation to 
generate situations with nonnormal distributions and dependency between 
effect sizes. Specifically, this effort tested the robustness of VG, as 
applied with the Raju et al . (1991) standard error of corrected correlations, 

to violations of the assumptions of independence and normality of primary 
data. Results of generations of 10,000 replications in 3,024 different 
combinations of conditions indicate that averaging correlations at the level 
of the primary study greatly underestimates the variance of P while skewness 
leads to overestimates of the variance of P. (Contains nine tables and seven 
figures. ) (Author/SLD) 
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Abstract 



Validity generalization meta-analysis (VG) examines the extent to which the validity of 
an instrument can be transported across settings. VG offers correction and summarization 
procedures designed in part to remove the effects of statistical artifacts on estimates of association 
between criterion and predictor. By employing a random effects model, the variability of a 
distribution, P, of population parameters, p, is estimated. When the variance of this distribution 
is estimated to be small, validity is said to generalize across situations. It is common for an 
admissible validity study to contribute more than one correlation to a meta-analysis. The.original 
VG meta-analysis (Pearlman, Schmidt, & Hunter, 1980) located 3,368 validity coefficients in 698 
studies. In addition, VG is often applied to instruments used to predict success on highly 
complex jobs. Such measures often have positively skewed distributions of predictor and 
criterion scores (Hunter, 1990). This effort employed Monte Carlo simulation to generate 
situations with non-normal distributions and dependency between effect sizes. Specifically, this 
effort tested the robustness of VG, as applied with the Raju et. al. (1991) standard error of 
corrected correlations, to violations of the assumptions of independence and normality of primary 
data. Results of generations of 10,000 replications in 3,024 different combinations of conditipns 
indicate that averaging correlations at the level of the primary study greatly underestimates the 
variance of P while skewness leads to overestimates of the variance of P. 




3 



The Robustness of the Standard Error of Summarized, Corrected Validity 
Coefficients to Non-Independence and Non-Normality of Primary Data 

While recent years have seen an enormous explosion of information dissemination 
technology and outlets for scholarly contribution, a family of new statistical methodologies 
has been developing as well. Meta-analysis has arisen in response to the need for quantitative 
methodology to systematically summarize the findings within a given area of inquiry while 
providing direction to policy makers who require input from the scientific community (Hedges 
& Olkin, 1985). 

Validity generalization meta-analysis, a specific application of meta-analytic techniques, 
examines the extent to which the validity of a specific instrument can be transported across 
settings through the use of summarized correlation coefficients. It tests situational specificity 
by examining the evidence for the validity of an instnunent with particular attention to 
whether the differences in sample validity coefficients result from error variance about a 
single population parameter (Hunter & Schmidt, 1990, Hunter, 1990). It seeks to examine the 
hypothesis of situational specificity for the instrument in question by estimating three 
quantities from data supplied by individual studies. The mean corrected sample correlation 
coefficient, the mean variance due to sampling error, and the variance due to differences 
between studies are estimated. The mean of the corrected sample correlation coefficients is 
interpreted as an estimate of the magnitude of association between the instrument and actual 
job performance (Schmidt & Hunter, 1977). By employing a random effects model, the 
variability of a distribution of population parameters is estimated. In short, the average of the 
error variance estimates for each individual primary study is used as an estimate of the 
sampling variability of the average of the validity coefficients. This quantity is subtracted 
from the variance between validity coefficients to arrive at an estimate of the variability 
between underlying parameters, p. When the variance of this distribution is estimated to be 
small, validity is said to generalize across situations. Since the error variance estimates are 
equal to the squared standard errors, the estimation of the standard error of the individual 
correlations obtained from individual validity studies is central to this process. This effort 
focused on the robustness of these estimates to non-independent and non-normal primary data. 

The field of industrial/organizational psychology has adapted as standard the concept of 
correcting correlations for statistical artifacts. Campbell (1990) has described validity 
generalization meta-analysis as an important methodology with critical implications for 
research and practice. He further indicates that the use of the random effects model with 
corrected validity coefficients has become the standard procedure in the field with the exact 
estimation procedures involved therein constituting the only area of remaining dispute. 
Resolution of the dispute will involve determinations about the relative accuracy of the 
competing estimation procedures as well as an examination of the extent to which the existing 
estimation procedures tolerate the actual data conditions of the typical validity generalization 
meta-analysis. The input data for a validity generalization meta-analysis, correlation 
coefficients obtained from primary validity studies, often do not meet the assumptions of the 
technique. The purpose of this effort is to extend the literature on the robustness of the 
standard error estimation procedures to violations of assumptions. 

The methodology of validity generalization meta-analysis has become such an important 
issue in education and psychology because it offers techniques that adjust for statistical 
artifacts such as sampling error. Examples of such statistical artifacts are sampling error, the 



unreliability of measurement, restriction of range, and lack of perfect construct validity. 
Validity generalization meta-analysis offers a set of methods, correction and summarization 
procedures, designed in part to remove the effects of statistical artifacts on estimates of the 
association between criterion and predictor. The successful application of these methods is, in 
theory, dependent upon the input data meeting specific statistical assumptions. A general 
assumption of meta-analysis as a class of procedures is independence of primary data. The 
concept of multiple effect sizes from a single study can be extended to validity generalization 
in that a single admissible validity study may contribute more than one correlation to a meta- 
analysis. A strict definition of the task of validity generalization limits meta-analysis to the 
study of statistically independent correlation coefficients (Hedges, 1989). The inclusion of a 
single correlation per study applies to the situation in which the meta-analysis is meant to 
accumulate evidence for the relationship between a specific selection measure and an accepted 
standard for the measurement of job performance. However, a complex job may require 
mastery or ability in a variety of areas, each of which is measured in a different way (Hunter, 
1990). The original meta-analysis in the area of validity generalization (Pearlman, Schmidt, 
& Hunter, 1980) located 3,368 validity coefficients in 698 studies. Contributing to the strong 
presence of non-independent correlations were the common occurrences of subgroup analyses, 
the use of multiple predictors, and the use of multiple criteria. 

Campbell (1990) has called for the field of industrial psychology to transcend reliance 
upon simple bivariate validity coefficients in favor of more sophisticated models of the 
antecedents of job performance. Doing so will require the design of studies that include 
multiple measures of both criterion and predictor. Such studies would contribute more than 
one correlation coefficient to a meta-analysis. 

Measures of performance for jobs involving high complexity or sales often have positively 
skewed distributions. Many individuals can not perform up to a reasonable standard of 
success, while a few individuals exhibit extremely high levels of performance (Hunter, 1990). 
The correction for restriction of range assumes equal conditional variance of the criterion, that 
is variance of the criterion is constant across all values of the predictor (Lord and Novick, 
1968). Furthermore, significance testing that employs the bivariate normal correlation model 
implies normality of criterion and predictor, and normal conditional probability distributions 
of the criterion, each with equal variance, given any value of the predictor (Neter, 
Wasserman, and Kutner, 1985). 

This effort provides a test of the robustness of the random effects model approach to 
validity generalization as it is applied with the Raju, Burke, Normand, and Langlois (1991) 
standard error of corrected correlations to the common violations of assumptions discussed 
above. Monte Carlo simulation has been employed to create situations in which assumptions 
are violated. The design matrix includes a condition in which normally distributed primary 
data are used and a condition in which a skewed distribution is created through the use of a 
chi square distribution with 4 degrees of freedom. The number of studies contained in each 
meta-analysis is varied across three levels: 5, 15, and 30. The number of subjects contained 
in each primary study is varied across three levels: 34, 67, and 100. The intercorrelation of 
criterion variables is varied across four levels: .00, .25, .50, and .75. The number of effect 
sizes, in this case corrected validity coefficients, that each study can contribute to a meta- 
analysis is varied across three levels: 1, 2, and 3. The method by which an effect size is 
selected from each primary study for inclusion in the meta-analysis is varied across three 
conditions: 1) a single effect size measure is randomly selected from the available set, 2) the 
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effect sizes from the primary study are averaged, and 3) all effect sizes are used in the meta- 
analysis. The value for u, the ratio of the standard deviation of criterion scores in the 
restricted range case to the unrestricted standard deviation, is calculated in three ways: 1) 

values were taken from a standard reference table based upon the properties of the normal 
distribution under specific pre-set range restriction parameters (Schmidt, Hunter, Pearlman, & 
, Shane, 1979), and 2) the ratio was calculated using two sample based strategies. All 
conditions were completely crossed by all other conditions except in the case of the cells that 
simulate primary studies containing only one effect size measure. In these cases, the effect 
size selection strategies are not applicable as the study can only contribute a single correlation 
coefficient to the meta-analysis. The matrix results in 42 different conditions occurring within 
each of 36 cells. 

Each cell was simulated according to the simulation strategies outlined in Appendix A of 
Raju et al. (1991). For each primary study simulated, this process involves sampling the 
reliability and restriction of range parameters from predetermined distributions. The 
distributions were selected to represent values found in typical meta-analyses in the field of 
personnel psychology (Callender & Osbum, 1980). The standard error formula under 
investigation has been slightly modified by changing an n to an n-1 since the Raju et al. 
(1991) article and the revised formula was utilized (Raju, personal communication, 1994). 
The SAS software package was utilized to generate 10,000 replications for each cell. Three 
conditions of homogeneity of effect size was simulated; V(P)=.00, .01, amd .04. These values 
represent the variance of the distributions of population validity coefficients from which the 
data were sampled. The mean of each distribution was set at p=.5. The resulting variance of 
P estimates produced by employing the random effects model could then be compared to .00, 
.01 and .04 respectively. 

The validity generalization meta-analysis procedure containing the random effects model 
approach and the Raju et al. (1991) standard error estimate performed very well across all 
conditions in which normally distributed data was simulated. This suggests that the technique 
is robust to non-independence of primary data when the data is sampled from normally 
distributed populations. This conclusion held across varying levels of intercorrelation of 
criterion variables and across varying calculation strategies. The skewed primary data 
conditions were consistently problematic as the triple correction procedure itself often 
overestimated p. This resulted in underestimates of the standard error of p'^, and 
consequently overestimates of V(P). It should be noted however, that these estimates were in 
line with what would be expected given the overestimates of p, suggesting that it is the triple 
correction procedure that is somewhat suspect when a skewed distribution is restricted. These 
finding must also be subjected to future efforts in which p is other than .5 and in which 
heterogeneity of effect size is present. Implications for the practitioner include caution in 
applying the triple correction to sample correlation coefficients when the data are skewed. 
Overestimates of p could result in inflated estimates of the individual test validities entering 
meta-analysis studies. In addition, these overestimates of validity result in overestimates of 
V(P). These overestimates could result in the interpretation of heterogeneity of effect size, or 
situational specificity, when it is not warranted. However, when the multiple correlations are 
averaged at the level of the primary study, V(P) is greatly underestimated. This could result 
in the conclusion of generalizability when in fact specificity may exist. 
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Table 1 . 

Cells in the Design 

rho = .5 rho=.5 rho=.5 n/ n/ 



V(P) = 0 


< 

T 

b 


V(P) = .04 


study 


meta 


rho yiyj 


Cell 1 






34 


5 


0.00 


Cell 2 


Cell 37 


Cell 55 


34 


5 


0.25 


Cells 


Cell 38 


Cell 56 


34 


5 


0.50 


Cell 4 


Cell 39 


Cell 57 


34 


5 


0.75 


Cells 






34 


15 


0.00 


Cell 6 


Cell 40 


Cell 58 


34 


15 


0.25 


Cell? 


Cell 41 


Cell 59 


34 


15 


0.50 


Celia 


Cell 42 


Cell 60 


34 


15 


0.75 


Cell 9 






34 


30 


0.00 


Cell 1 0 


Cell 43 


Cell 61 


34 


30 


0.25 


Cell 1 1 


Cell 44 


Cell 62 


34 


30 


0.50 


Cell 12 


Cell 45 


Cell 63 


34 


30 


0.75 


Cell 13 






67 


5 


0.00 


Cell 14 


Cell 46 


Cell 64 


67 


5 


0.25 


Cell 15 


Cell 47 


Cell 65 


67 


5 


0.50 


Cell 16 


Cell 48 


Cell 66 


67 


5 


0.75 


Cell 17 






67 


15 


0.00 


Cell 18 


Cell 49 


Cell 67 


67 


15 


0.25 


Cell 19 


Cell 50 


Cell 68 


67 


15 


0.50 


Cell 20 


Cell 51 


Cell 69 


67 


15 


0.75 


Cell 21 






67 


30 


0.00 


Cell 22 


Cell 52 


Cell 70 


67 


30 


0.25 


Cell 23 


Cell 53 


Cell 71 


67 


30 


0.50 


Cell 24 


Cell 54 


Cell 72 


67 


30 


0.75 


Cell 25 






100 


5 


0.00 


Cell 26 






100 


5 


0.25 


Cell 27 






100 


5 


0.50 


Cell 28 






100 


5 


0.75 


Cell 29 






100 


15 


0.00 


Cell 30 






100 


15 


0.25 


Cell 31 






100 


15 


0.50 


Cell 32 






100 


15 


0.75 


Cell 33 






100 


30 


0.00 


Cell 34 






100 


30 


0.25 


Cell 35 






100 


30 


0.50 


Cell 36 






100 


30 


0.75 
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Table 2. 

Conditions Within Each Celt 

es selection 



Condition u 

1 T 

2 1 

3 1 

4 1 

5 1 

6 1 

7 1 

8 1 

9 1 

10 1 

11 1 

12 1 

13 1 

14 1 

15 2 

16 2 

17 2 

18 2 

19 2 

20 2 

21 2 

22 2 

23 2 

24 2 

25 2 

26 2 

27 2 

28 2 

29 3 

30 3 

31 3 

32 3 

33 3 

34 3 

35 3 

36 3 

37 3 

38 3 

39 3 

40 3 

41 3 

42 3 



method nes 

na 1 

1 2 

2 2 

3 2 

1 3 

2 3 

3 3 

na 1 

1 2 

2 2 

3 2 

1 3 

2 3 

3 3 

na 1 

1 2 

2 2 

3 2 

1 3 

2 3 

3 3 

na 1 

1 2 

2 2 

3 2 

1 3 

2 3 

3 3 

na 1 

1 2 

2 2 

3 2 

1 3 

2 3 

3 3 

na 1 

1 2 

2 2 

3 2 

1 3 

2 3 

3 3 



distribution 
normal 
normal 
normal 
normal 
normal 
normal 
normal 
chi square 
chi square 
chi square 
chi square 
chi square 
chi square 
chi square 
normal 
normal 
normal 
normal 
normal 
normal 
normal 
chi square 
chi square 
chi square 
chi square 
chi square 
chi square 
chi square 
normal 
normal 
normal 
normal 
normal 
normal 
normal 
chi square 
chi square 
chi square 
chi square 
chi square 
chi square 
chi square 




u1 = pop./pop., u2=sample/pop.. u3=sample/sample. 
es method 1 = randomly pick. 2= average. 3 = use ail. 
nes = number of effect sizes. 



Table 3 

Distnbution of Situabon Mean V(P) Estimates 
By Effect Size Selection Method and V(P) Condition. 



Combination of Effect Size Selection Method and V(P) Condition 



Ouartiles 


picK/.OO 


pick/.Ol 


piCk/.04 


avg/.OO 


avg/.OI 


avg/.04 


all/.OO 


all/.oi 


all/.04 


Min 


-0.000590 


0.006724 


0.036267 


-0.076246 


-0.056591 


-0.025653 


-0.006450 


0.001964 


0.025752 


25th 


0.001869 


0.011381 


0.039007 


-0.024871 


-0.019560 


0.009104 


0.001526 


0.010210 


0.035139 


50th 


0.006743 


0.014669 


0.040231 


-0.012316 


-0.003693 


0.022664 


0.005299 


0.013517 


0.038317 


75th 


0.009059 


0.022956 


0.046994 


-0.001525 


0.009845 


0.034146 


0.008874 


0.021146 


0.045747 


Max 


0.015230 


0.028349 


0.055974 


0.005625 


0.020016 


0.049192 


0.015066 


0.027535 


0.054963 


Mean 


0.006026 


0.01 6987 


0.043409 


-0.016545 


-0.006680 


0.019796 


0.005464 


0.015246 


0.039964 


S.D. 


0.004132 


0.005969 


0.006161 


0.018592 


0.019295 


0.017814 


0.004312 


0.006270 


0.006616 


No. of Situations 


432 


216 


216 


432 


216 


216 


432 


216 


216 


Max error as % of: 




















V(P). .01 or .04 




163.49% 


39.94% 




665.91% 


164.63% 




1 75.35% 


37.46% 


V(P),q=.5,‘large‘ 


46.56% 


56.10% 


46.84% 


242.30% 


203.60% 


201.34% 


46.06% 


53.61% 


45.61% 



Each Situation consists of a condition occurring within a cell and represents 
the mean of 10,000 replications. Only those conditions with 2 or 3 effect 
sizes per primary study are included. 
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Means for V(P) Estimates 

By Effect Size Selection Method & V(P) 




rH 



(0 

)( 




o 




S0ieuJ!iS3 (d)A ue0i/\| 



pick — average 



Table 4 

Distribution of Situation Mean V(P) Estimates 
By Distribution Type and V(P) Condition, 



Combination of Distribution Type and V(P) Condition 



Quartiles 


norm-,00 


norm-,01 


norm-,04 


chisq-.OO 


chisq-.OI 


chisq-.04 


Min 


-0.005521 


0.002903 


0,026359 


0.008497 


0.020093 


0.041309 


25th 


0.001435 


0,010441 


0.037062 


0.010969 


0,025228 


0.049914 


50th 


0,001744 


0,010911 


0,038450 


0.011810 


0.026034 


0.053588 


75th 


0.002012 


0,011391 


0,039304 


0.013258 


0.026600 


0.055687 


Max 


0,004427 


0.012214 


0,040285 


0.015230 


0.028349 


0.066166 


Mean 


0,001613 


0.010523 


0,037646 


0.012111 


0.025785 


0.054301 


S.D. 


0,001038 


0.001597 


0.002597 


0,001545 


0.001591 


0.006399 


No. of Situations 


180 


90 


90 


180 


90 


90 


Max error as % of: 
V(P), .01 or .04 
V(P),q=.3;’medium" 


44.78% 


70.97% 

57.56% 


34.10% 

110.64% 


123.53% 


183.49% 

148.83% 


65.42% 

212.23% 



Each Situation consists of a condition occurring within a cell and represents 
the mean of 10,000 replications. Only u3 conditions where non-averaging 
effect size selection methods were used are Included. 
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Means for V(P) Estimates 

By Distribution Type and V(P) 



▼H 





S81BUJIIS3 (d)A ueei/M 



normal chi-square 



Tabl« 5 

Oiatribution of Situation Mean V(P) Eatimatea 
By u Calculation Method and V(P) Condition. 



Combination of u Calculation Method and V(P) Condition 



Quanilea 


ul/.OO 


ul/.OI 


U1/.04 


u2/.00 


u2/.01 


u2/.04 


U3/.00 


U3/.01 


U3/.04 


Min 


•0.002552 


0.005659 


0.026623 


•0.006450 


0.001964 


0.025752 


•0.005521 


0.002903 


0.026359 


2Sth 


0.002604 


0.011765 


0.036350 


0.000734 


0.009283 


0.036067 


0.001366 


0.010156 


0.036762 


50th 


0.003049 


0.012562 


0.039671 


0.001142 


0.010069 


0.037666 


0.001699 


0.010839 


0.036186 


75th 


0.004375 


0.013656 


0.040473 


0.001431 


0.010695 


0.036745 


0.001981 


0.011314 


0.039230 


Max 


0.007151 


0.014669 


0.041616 


0.003463 


0.011703 


0.040470 


0.004427 


0.012214 


0.040285 


Mean 


0.003351 


0.012394 


0.036794 


0.000928 


0.009622 


0.036683 


0.001564 


0.010343 


0.037316 


S.O. 


0.001420 


0.001766 


0.002729 


0.001161 


0.001766 


0.002836 


0.001121 


0.001725 


0.002784 


No. of Situationa 


144 


72 


72 


144 


72 


72 


144 


72 


72 


Max error as % of; 
V(P), .01 or .04 
V(P),q= .3, "medium" 


56.00% 


46.69% 

36.03% 


28.44% 

92.28% 


52.32% 


60.36% 

65.16% 


35.62% 

115.56% 


44.78% 


70.97% 

57.56% 


34.10% 

110.84% 



Each Situation conaiata of a condition occurring within a ceii and repreaenta 
the mean of 10.000 repiicationa. Oniy normally diatributad corrditiona 
where nomaveraging effect size selection rrtethoda were used em included. 
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Table 6 

Dittribution of Situation Mean V(P) Eatimataa 
By Intarcorratation of Criterion Variabiaa and Vff*) Condition. 



Combination of Intarcorrelation of Criterion Variabtea artd V^) Condition 



Ouartitea 


.OO/.OO 


.25/.00 


.25/.01 


.2S/.04 


Min 


0.000458 


0.000621 


0.006542 


0.032612 


2Sth 


0.001653 


0.001466 


0.010611 


0.037416 


50th 


0.001974 


0.001673 


0.011119 


0.036566 


75th 


0.002256 


0.001664 


0.011404 


0.039166 


Max 


0.004427 


0.003101 


0.012214 


0.040106 


Mean 


0.002133 


0.001707 


0.010960 


0.036002 


S.D. 


0.000649 


0.000422 


0.000771 


0.001695 


No. of Situatlona 


45 


45 


30 


30 


Max error aa % of: 










V(P), .01 or .04 






22.14% 


18.47% 


V(P),q= .3,"medium“ 


35.91% 


25.15% 


17.96% 


59.92% 



.50/.00 


.50/.01 


.50/.04 


.75/.00 


.75/.01 


.75/.04 


-0.001361 


0.005769 


0.030613 


-0.005521 


0.002903 


0.026359 


0.001366 


0.010276 


0.036926 


0.000999 


0.010149 


0.036632 


0.001569 


0.011009 


0.038431 


0.001462 


0.010792 


0.036069 


0.001935 


0.011279 


0.039370 


0.001635 


0.011179 


0.039211 


0.003231 


0.012076 


0.040285 


0.002730 


0.011943 


0.040231 


0.001577 


0.010499 


0.037710 


0.001035 


0.010089 


0.037227 


0.000612 


0.001430 


0.002404 


0.001561 


0.002185 


0.003319 


45 


30 


30 


45 


30 


30 




42.11% 


22.97% 




70.97% 


34.10% 


26.21% 


34.16% 


74.52% 


44.76% 


57.56% 


1 10.64% 



Each Situation conaiata of a condition occurring within a cell and repreeenta 
the mean of 10,000 replicationa. Only normally diatributed. u3 conditiona 
where non-averaging effect aize aelection methodawere uaed are included. 
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Table 7 

Distribution of Situation Mean V(P) Estimates 
By Sample Size of Primary Study and V(P) Condition. 



Combination of Sample Size of Primary Study and V(P) Condition 



Quartiles 


34/.00 


34/.01 


34/.04 


67/.00 


67/.01 


67/. 04 


100/. 00 


Min 


-0.005521 


0.002903 


0.026359 


-0.001446 


0.005851 


0.031443 


-0.000365 


25th 


0.001077 


0.010276 


0.036795 


0.001367 


0.010630 


0.037650 


0.001482 


50th 


0.001839 


0.010746 


0.038026 


0.001693 


0.011142 


0.039211 


0.001706 


75th 


0.002281 


0.011179 


0.038586 


0.001981 


0.011472 


0.039513 


0.001891 


Max 


0.004427 


0.012214 


0.039606 


0.003355 


0.012076 


0.040285 


0.002290 


Mean 


0.001554 


0.010235 


0.037044 


0.001644 


0.010811 


0.038249 


0.001641 


S.D. 


0.001570 


0.001889 


0.002836 


0.000774 


0.001192 


0.002204 


0.000449 


No. of Situations 


60 


45 


45 


60 


45 


45 


60 


Max error as % of: 
V(P), .01 or .04 
V(P),q=. 3, "medium" 


44.78% 


70.97% 

57.56% 


34.10% 

110.64% 


27.21% 


41.49% 

33.65% 


21.39% 

69.41% 


18.57% 



Each Situation consists of a condition occurring within a cell and represents 
the mean of 10,000 replications. Only normally distributed, u3 conditions 
where non>averaging effect size selection methods were used are included. 
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Table 6 

Distribution of Situation Mean V(P) Estimates 
By Sample Size per Mela*Analysis and V(P) Condition. 



Combination of Sample Size per Primary Study and V(P) Condition 



Quartiles 


5/.00 


5/.01 


5/.04 


15/00 


15/01 


15/.04 


30/.00 


30/01 


30/.04 


Min 


-0.005521 


0.002903 


0.026359 


0.000126 


0.008621 


0.034769 


0.001101 


0.010149 


0.036613 


25th 


0.000963 


0.006542 


0.033356 


0.001463 


0.010441 


0.037418 


0.001644 


0.010911 


0.038022 


50th 


0.001616 


0.010611 


0.038329 


0.001656 


0.010697 


0.038771 


0.001801 


0.01 1 1 57 


0.038203 


75th 


0.002176 


0.011179 


0.039238 


0.001946 


0.011345 


0.039246 


0.001997 


0.011452 


0.039325 


Max 


0.004427 


0.012214 


0.040285 


0.003101 


0.011960 


0.040106 


0.002912 


0.011043 


0.039854 


Mean 


0.001329 


0.009659 


0.036289 


0.001670 


0.010715 


0.038241 


0.001841 


0.011194 


0.038409 


S.D. 


0.001649 


0.002402 


0.003893 


0.000551 


0.000777 


0.001320 


0.000332 


0.000416 


0.000964 


No. of Situations 


60 


30 


30 


60 


30 


30 


60 


30 


30 


Max error as % of: 




















V(P). .01 or .04 




70.97% 


34.10% 




19.60% 


13.08% 




19.43% 


8.47% 


V(P),q=.3,"medium" 


44.78% 


57.56% 


110.64% 


25.15% 


15.90% 


42.43% 


23.62% 


15.76% 


27.47% 



Each Situation consists of a condition occurring within a cell and represents 
the mean of lO.CXX) replications. Only normally distributed, u3 conditions 
where non-averaging effect size selection methods were used are included. 
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Table 9 

Distribution of Situation Mean V(P) Estimates 
By Effect Sizes per Primary Study and V(P) Condition. 



Combination of Effect Sizes per Primary Study and V(P) Condition 



Quartiles 


1/.00 


1/.01 


1/.04 


2/.00 


2/.01 


2/.04 


3/.00 


3/.01 


3A04 


Min 


-0.000701 


0.010276 


0.037547 


-0.004527 


0.004570 


0.028458 


-0.005521 


0.002903 


0.026359 


25th 


0.001571 


0.010911 


0.036147 


0.001364 


0.010360 


0.036913 


0.001296 


0.009939 


0.036613 


50th 


0.001626 


0.011279 


0.039106 


0.001695 


0.011016 


0.038156 


0.001742 


0.010644 


0.036166 


75th 


0.002056 


0.011472 


0.039541 


0.002012 


0.011361 


0.039238 


0.001969 


0.01 1 1 79 


0.03921 1 


Max 


0.002766 


0.012062 


0.040151 


0.003660 


0.012214 


0.040231 


0.004427 


0.01 2076 


0.040265 


Mean 


0.001329 


0.009659 


0.036269 


0.001670 


0.010715 


0.036241 


0.001841 


0.011194 


0.036409 


S.D. 


0.001649 


0.002402 


0.003693 


0.000551 


0.000777 


0.001320 


0.000332 


0.000416 


0.000964 


No. of Situations 


36 


16 


16 


72 


36 


36 


72 


36 


36 


Max error as % of: 
V(P), .01 or .04 
V(P) ,q =3 .3,“medium“ 


22.60% 


20.62% 

16.72% 


6.13% 

19.90% 


36.72% 


54.30% 

44.04% 


26.86% 

93.62% 


44.78% 


70.97% 

57.56% 


34.10% 

110.64% 



Each Situation consists of a condition occurring within a cell and represents 
the mean of 10,000 replications. Only normally distributed, u3 conditions 
where non-averaging effect size selection methods were used are included. 
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